However, spurious correlation remains an ongoing challenge, primarily due to the difficulty in precisely detecting these samples.

Thesebiased models suffer from the poor generalization capability when evaluated on unbiased datasets. Existing approaches for debiasing often identify and emphasize those samples withnosuchcorrelation (i.e.,bias-conflicting)without defining the bias type in advance. However, such bias-conflicting samples are significantly scarce in biased datasets, limiting the debiasing capability of these approaches.

Whileprevious work tackles this issue by using explicit labeling on the spuriously correlated attributes or presuming a particular bias type, we instead utilize a cheaper, yet generic form of human knowledge, which can be widely applicable to various types ofbias.

However, ERM has been known to cause a learned classifier to be biased toward spurious correlations between predefined classes and latent attributes that appear in a majority of training data [12].

For real-world visual datasets (like facial dataset or ImageNet), the unavailability of strict counterfactual data is a common challenge.